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INTRODUCTION 


Enhancing  the  capability  of  U.S.  forces  to  detect,  locate,  attack,  and  destroy 
theater  missiles  in  all  phases  of  conflict  and  theaters  of  operation  has  become  a  high 
defense  priority  since  Operation  Desert  Storm.  Recently,  the  Theater  Missile  Defense 
Automatic  Target  Recognition  (TMD-ATR)  and  Rapid  Response  Targeting  Against 
Mobile  Ground  Targets  (RTM)  programs  merged  and  conducted  a  joint  effort  to  explore 
technologies  to  advance  this  capability.  The  effort  involved  the  use  of  integrated 
advanced  synthetic  aperture  radar  (SAR)  and  ATR  technologies,  the  efficacy  of  which 
was  tested  in  a  series  of  target  detection  demonstration  flights.  Specifically,  real-time 
ATR  algorithms  developed  by  Sandia  National  Laboratories  were  integrated  with 
enhanced  SAR  (ESAR)  sensor  imagery  in  order  to  investigate  the  ability  of  operators 
aboard  the  E-8C  Joint  Surveillance  Target  Attack  Radar  System  (JSTARS)  aircraft  to 
detect,  locate,  and  identify  mobile  missile  launchers  on  the  ground. 

Despite  recent  advances  in  ATR  systems,  there  is  consensus  that  the  human 
operator  will  remain  “in-the-loop”  as  an  integral  part  of  near  and  mid-term  target 
acquisition  systems  (Kuperman,  1997;  Aerospace  Daily,  1996).  The  consideration  of 
human  system  interface  (HSI)  issues  is  critical  to  further  development  and  integration  of 
these  systems.  In  this  regard,  a  HSI-oriented  human  factors  evaluation  of  the  ESAR/ ATR 
integration  concept  was  conducted  in  support  of  the  ESAR/ ATR  integration  concept.  The 
following  report  documents  the  results  of  this  evaluation.  The  human  factors  assessment 
was  based  on  task  analysis  results,  flight  crew  questionnaire  responses,  and  the  direct 
observation  of  flight  crew  performance  by  a  human  factors  engineer,  who  participated  in 
an  1 1-hour  demonstration  flight.  The  evaluation  objective  was  to  establish  guidelines  for 
the  integration  of  ATR  and  ESAR  within  JSTARS. 
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HUMAN  FACTORS  EVALUATION 

The  primary  objectives  of  the  human  factors  evaluation  of  the  ESAR/ATR 
integration  were  to: 

•  Assess  the  target  prosecution  benefits  of  the  integration  of  ATR  information 
(target  identification  [ID],  cue  box,  ATR  results  summary)  with  ESAR 
imagery 

•  Identify  user  requirements  for  ATR  information  using  input  from  experienced 
Joint  Test  Force  (JTF)  personnel  who  operated  the  system  prototype 

•  Provide  HSI  design  guidelines  and  recommendations  for  future  ESAR  /ATR 
integration  efforts 

The  objectives  outlined  above  were  accomplished  using  the  following  techniques: 

I.  Image  analyst  (IA)  task  analysis 

II.  JTF  operator  post-demonstration  flight  questionnaire 

IE.  In-flight  observations  and  guidelines  review  by  a  human  factors  engineer 

I.  Image  Analyst  Task  Analysis 

The  task  analysis  was  performed  during  crew  familiarization  and  by  interviews 
with  two  JTF  image  analysts.  The  task  analysis  consisted  of  a  simple  task  decomposition 
procedure.  The  task  decomposition  served  to  delineate  the  specific  tasks  required  of  the 
ESAR/ATR  workstation  operator  in  order  to  facilitate  the  definition  of  specific 
information  display  and  control  requirements.  Definition  of  these  requirements  is  useful 
in  determining  practical  methods  of  integrating  ATR  with  ESAR  imagery.  A  synopsis  of 
the  task  analysis  results  showing  the  primary  tasks  and  information  requirements  are 
listed  in  Table  1.  In  short,  the  tasks  of  detecting,  identifying,  designating,  and  handing- 
off  a  designated  target  were  accomplished  in  several  steps,  beginning  with  the  use  of  the 
ATR  software  to  retrieve  the  SAR  image  file.  The  image  was  then  overlaid  with  the  ATR 
to  identify  and  designate  the  Transporter-Erector-Launcher  (TEL).  This  information  was 
then  available  for  dissemination  to  the  fighter  by  way  of  verbal  report.  It  should  be  noted 
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that  the  ATR  Tool  was  not  specifically  designed  to  support  the  imagery  analyst  in  target 
acquisition  tasks,  but  rather  to  retrieve  and  save  images,  manage  files,  and  to  modify  the 
ATR  information.  The  ATR  Tool  display  was  also  totally  unrelated  to  the  existing 
JSTARS  graphics  display  and  its  associated  functions. 

Table  1:  JTF  Workstation  Operators  Tasks/Subtasks  and  Information/Control 

Requirements 


JTF  Workstation  Operator 
Tasks/Subtasks 

•  .ESAR  Image  File  Management 

♦  Retrieve  Image  Files 

*  View  /  Assess  Quality  of 
Images 

•  Store  Image  Files 

♦  ESAR  Image  Manipulation 

Adjust  Brightness  /  Contrast 

•  Zoom 


•  Visual  Search  and  Target  Identification 
•  Assess  Image  Qual  ity 
»  Identify  Scene  Objects 


•  Pass  Information  to  Ground  Agencies  / 
Fighter 

•  Determine  location  coordinates 

•  Cut/paste  target  chips 

•  Save  Images 
Send  Images 


Information  and  Control  Requirements 

JSTARS  Graphics  Display 
ATR  T ool  Software  Interface 
Available  Files  Listing 
SAR  Image 
Input  Device 


JSTARS  Graphics  Display 
Pull  down  Menus 
SAR  Image 

Toolbar  of  Image  Editing  Options 
Input  Device 


JSTARS  Graphics  Display 
Pull  down  Menus 
SAR  Image 

Toolbar  of  Image  Editing  Options 
ATR  Target  ID  information 
Target  location  coordinates 


JSTARS  Graphics  Display 
Pull  down  Menus 
SAR  Image 

Toolbar  of  Image  Editing  Options 
ATR  Target  ID  information 
Target  location  coordinates 


Figure  1  below  shows  the  workstation  configuration  aboard  the  E8-C  JSTARS. 


Figure  1:  JSTARS  Operator  Workstations 


II.  JTF  Operators  Questionnaire 

A  post-flight  questionnaire  was  completed  by  four  JTF  aircrew,  each  of  whom 
operated  the  JSTARS  graphics  display  workstation  during  ESAR/ATR  data  collection 
demonstration  flights.  The  questionnaire  was  designed  to  collect  operator  input  in  the 
areas  of: 

•  ATR  Information  (target  descriptions  and  cue  boxes) 

•  ESAR  E-4  and  E-8  Imagery  (representing  medium  and  high  resolution, 
respectively) 

•  Human-Machine  Interface 
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Three  of  the  four  operators  had  prior  SAR  experience  on  the  E-8  JSTARS 
platform,  ranging  from  2  to  3  years  and  a  total  of  50  to  500  hours.  Two  respondents 
defined  their  current  crew  position  as  surveillance,  another  as  weapons  operator,  and  the 
fourth  as  an  Aerial  Observer  Technician  (AOT).  Tabulated  questionnaire  responses  may 
be  found  in  Appendix  A. 

ATR  Information 

The  ATR  information  consisted  of  the  color-coded  boxes,  text  or  symbols  that  related  to 
the  ATR  designations.  The  information  was  manifested  across  five  information  types:  a 
Target  Description  which  identified  the  missile  orientation  on  the  TEL  (i.e.,  ‘TEL  UP’ 
signified  the  missile  was  erect  and  in  launch  configuration,  ‘TEL  DOWN’  indicated  the 
missile  in  stowed  position,  while  ‘TEL  OFF’  indicated  the  absence  of  a  missile  on  the 
TEL);  Target  Coordinate  information,  which  provided  the  x,  y  pixel  location  of  the  target 
on  the  SAR  image;  ATR  Cue  Boxes  which  enclosed  the  object  identified  by  the 
autoclassifier;  an  ATR  Results  Summary,  an  optional  window  selectable  by  the  operator, 
which  portrayed  the  probability  value  assigned  to  the  designated  object;  and  an  ATR 
Color  Coding  option,  which  permitted  the  operator  to  specify  the  color-coding  of  the 
displayed  ATR  information. 

Results  of  the  questionnaire  indicated  that  all  operators  found  the  target 
identification  description  (e.g.,  TEL  UP)  useful  in  aiding  target  acquisition  and 
identification.  Three  operators  stated  that  the  target  location  coordinates  should  be 
presented  adjacent  to  the  target  ID  label  on  the  ATR  overlay.  Similarly,  all  operators 
found  the  overlaid  ATR  information  useful  in  aiding  target  prosecution  and  none  reported 
that  it  cluttered  the  display.  Two  operators  commented  that  the  ATR  cue  boxes  enabled 
them  to  quickly  find  and  identify  both  real  and  potential  targets.  The  additional  display  of 
the  confidence  rating  for  the  ATR  declaration  was  cited  as  a  needed  improvement  by 
three  operators. 

The  impact  of  the  five  types  of  ATR  information  was  also  assessed  in  regard  to 
situational  awareness  (SA)  and  workload  using  a  five  point  rating  scale.  The  lower  end 
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of  the  scale  (‘1’  or  ‘2’)  indicated  that  the  ATR  information  had  a  negative  impact  on 
performance,  such  as  reducing  situational  awareness  or  increasing  workload;  mid-scale  (a 
value  of  ‘3’)  represented  a  neutral  stance,  whereby  the  ATR  information  neither 
increased  or  decreased  SA  or  workload;  while  the  high  end  (‘4’  and  ‘5’)  represented 
positive  impacts  of  the  ATR  information  on  increasing  situational  awareness  or  reducing 
workload.  Four  operators  rated  the  impact  on  SA,  of  whom  three  also  responded  to  the 
question  on  impact  on  workload.  The  average  ratings  obtained  for  SA  and  workload  are 
presented  in  Tables  2  and  3  below. 

Table  2:  Average  Ratings  for  Impact  of  Type  of  ATR  Information  on  SA 

Type  of  ATR  Target  '  Target  ATR  Cue  ATR  Results  ATR  Color 

Information  Description  Coordinates  Boxes  Summary  Coding 

Impact  on  SA  4.5  4.5  '  4.75  4  4.75 

SA:  1  =  Significantly  Negative  3  =  None  5  =  Significantly  Positive 

Table  3:  Average  Ratings  for  Impact  of  Type  of  ATR  Information  on  Workload 

Type  of  ATR  Target  Target  ATR  Cue  ATR  Results  ATR  Color 

Information  Description  Coordinates  Boxes  Summary  Coding 

Impact  on  2.7  2.7  3.7  :  -  :  2:3  3.3 

Workload 

Workload:  1  -  Significant  Increase  3  =  None  5  =  Significant  Decrease 

In  summary,  all  four  operators  rated  the  ATR  information  as  useful  and  as  having 
a  positive  impact  on  SA.  Of  the  three  respondents  who  also  rated  workload,  two 
responded  that  the  ATR  cue  boxes  and  color  coding  “somewhat  decreased”  workload. 
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The  other  types  of  ATR  information,  on  average,  were  rated  as  having  no  impact  on 
workload. 


ESAR  E-4  and  E-8  Imagery 

While  both  E-4  and  E-8  imagery  were  available  during  the  mission,  only  one  of  the  four 
operators  actually  used  E-4  imagery;  however,  that  operator  rated  it  as  having  a 
“somewhat  positive”  effect  in  aiding  target  acquisition  and  identification.  All  four 
operators  rated  the  E-8  imagery  useful  in  aiding  target  acquisition,  and  each  one  reported 
a  noticeable  enhancement  in  resolution  over  standard  SAR  imagery.  The  ESAR  E-8 
imagery  was  rated  as  having  a  positive  impact  on  increasing  SA  (subject  responses  split 
between  “somewhat  positive”  and  “significantly  positive”).  No  one  reported  the  E-8 
imagery  as  having  any  additional  effect  on  visual  fatigue. 

Human  Machine  Interface 

Each  operator  rated  the  HMI  software  developed  for  the  demonstration  (the  ATR  Tool ) 
with  respect  to  impact  on  workload,  its  effect  on  image  handling  and  targeting  operations, 
and  the  overall  value  it  added  to  the  integrated  system  capabilities.  It  should  be  noted  that 
the  ATR  Tool  was  adapted  from  Khoros  (C+)  as  a  developmental  tool  for  use  by  Sandia 
National  Laboratory  engineers  and  the  Northrop  Grumman  integration  team. 

All  four  operators  reported  that  the  HMI  was  easy  to  learn  and  understand.  With 
regard  to  the  HMI’s  impact  on  workload,  two  rated  it  as  having  a  “somewhat  negative” 
effect  by  increasing  workload,  one  reported  a  “somewhat  positive”  effect  by  reducing 
workload,  and  the  fourth  reported  “no  effect”  on  workload.  Two  operators  assessed  the 
overall  value  of  the  HMI  as  “somewhat  positive,”  while  two  rated  it  as  having  a 
“somewhat  negative”  value.  Image  handling  problems  that  were  noted  consisted  of  the 
loss  of  target  location  coordinates  when  cropping  a  portion  of  an  image  to  save,  and  the 
loss  of  the  ESAR  display  when  trying  to  view  the  normal  graphics  display.  One  operator 
commented  that  refreshing  the  menu  to  update  the  SAR  file  listing  was  a  feature  he 
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disliked.  Three  operators  commented  that  the  ESAR/ATR  capability  should  be  integrated 
into  the  JSTARS  graphic  display  instead  of  operating  as  a  stand-alone  system. 

III.  Observations,  Established  Guidelines  and  Recommendations 

Displays  used  in  tactical  and  command  and  control  operations  must  be 
designed  to  aid  the  user  in  developing  and  maintaining  good  SA  (Walrath,  1996).  There 
are  several  general  and  specific  design  guidelines  available  to  aid  in  the  design  of 
workstation  displays  and  controls.  The  two  primary  guidelines  used  for  the  current  effort 
were  Salvendy’s  1987  Handbook  of  Human  Factors,  and  the  1989  MIL-STD-1472D, 
Human  Engineering  Design  Criteria  for  Military  Systems,  Equipment  and  Facilities. 
Walrath  (1996)  also  provides  relevant  guidelines  and  considerations  for  the  display  of 
information.  (The  guidelines  and  recommendations  discussed  in  the  following 
paragraphs  refer  to  the  ATR  Tool  developed  by  the  ESAR/ATR  integration  team.) 

ATR  Color  Coding 

The  use  of  color  coding  is  helpful  when  an  operator  must  search  for  specified  information 
and  symbol  density  is  high  (Salvendy,  1987).  The  HMI  prototype  allowed  the  operator  to 
selectively  color-code  the  ATR  cue  boxes  at  each  level  of  detection:  (1)  Focus  of 
Attention  (FOA),  (2)  Second  Level  Detection  (SLD),  and  (3)  ID  (cue  boxes  and 
associated  target  identification  labels).  It  should  be  noted  that  these  levels  corresponded 
to  the  three  levels  of  processing  conducted  by  the  ATR  algorithm  in  classifying  a  TEL, 
not  to  the  operator’s  task.  Furthermore,  it  was  only  when  the  image  ‘passed’  at  each 
level,  that  its  classification  was  reported.  Although  the  operators  generally  favored  the 
option  to  select  preferred  colors  for  displaying  this  information,  a  single,  consistent  color 
scheme  for  all  workstations  is  recommended.  MIL-STD-1472D  (1989)  states  that 
consistent,  meaningful  codes  should  be  used  for  user-computer  interface  display  coding. 
The  use  of  a  consistent  color  scheme  reduces  the  probability  of  misidentification  errors, 
especially  when  operators  are  viewing  multiple  workstations  and  SAR  images.  Red, 
yellow,  and  green  (or  white)  typically  are  used  to  indicate  specific  conditions.  Red 
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signals  the  information  is  of  immediate  attention,  yellow  indicates  caution,  and  green  (or 
white)  denotes  fully  operational  or  satisfactory.  It  is  recommended  that  these  colors  be 
used  when  coding  ATR  information.  Redundant  coding  (distinctive  cue  box  shapes)  of 
ATR  information  is  also  recommended  ( M1L-STD-1472D ,  1989;  Silverstein,  1987).  For 
example,  the  ID  level  detections  should  be  cued  using  a  symbol  that  is  different  from 
those  used  for  the  two  other  detection  levels. 

Target  Description  Labeling 

The  ATR  information  contained  target  ID  labels  denoting  target  name  and  condition  (e.g., 
TEL  DOWN).  It  was  observed  during  the  demonstration  flights  that,  when  overlaid  on 
the  SAR  image  (presented  in  gray  scale),  red  text  was  more  difficult  to  read  compared  to 
green,  yellow,  or  white  text.  This  is  probably  attributable  to  differences  in  CRT 
luminance  levels  as  a  function  of  color.  CRT  output  luminances  are  typically  brighter  for 
green,  blue,  and  white.  Ambient  illumination  (i.e.,  overhead  lighting)  also  can  affect  the 
luminance  contrast  and  apparent  color  of  information  displayed  on  the  CRT  (Silverstein, 
1987).  Therefore,  it  is  recommended  that  if  red  is  used  as  a  color  for  target  labeling,  both 
letter  size  and  brightness  level  should  be  increased.  MIL-STD-1472D  (1989,  para. 
5.5.5.14  -  5.5.5.15)  also  provides  guidelines  for  determining  display  symbol  and  character 
height  as  a  function  of  viewing  distance.  Future  ESAR/ATR  integration  should  be 
developed  with  careful  attention  to  these  guidelines  to  ensure  text  readability. 

ATR  Information  Selectabilitv 

Additionally,  it  was  recommended  that  the  overlaid  ATR  information  be  made  selectable 
or  de-selectable  using  a  simple  ON/OFF  (toggle)  switch  displayed  as  part  of  the  menu. 

Toolbars 

As  revealed  in  the  task  analysis,  there  are  several  repeated  image  manipulation  and  file 
retrieval  operations  required  of  the  workstation  operator  when  performing  an  activity. 

The  existing  software  interface  is  keystroke  intensive,  requiring  users  to  input  a  sequence 
of  commands  and  employ  a  series  of  menus  in  order  to  open  windows  to  accomplish 
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simple  tasks.  Linking  the  associated  operations  and  displaying  each  task  or  activity  as  an 
icon  on  a  10  -  12  item  toolbar  would  allow  easier  and  quicker  access  to  these  functions. 


Content  of  ATR  Information 

The  ATR  information  overlay  consisted  of  color-coded  cue  boxes  and  target  ID  labels 
(e.g.,  TEL  UP,  TEL  DOWN)  for  the  ID  level  of  the  ATR  algorithm  output.  During  the 
flight  demonstration  and  crew  familiarization,  several  operators  noted  that  the  inclusion 
of  height,  width,  and  length  information  for  those  targets  with  TEL  ID  declarations  would 
be  useful.  The  JTF  operators  also  desired  a  simple  numeric  ATR  confidence  rating  (e.g., 
0.80)  to  be  displayed  along  with  the  ATR  output. 


CONCLUSIONS  AND  RECOMMENDATIONS 

This  evaluation  was  conducted  to  assess  the  utility  of  the  integration  of  ATR 
information  and  ES  AR  imagery  as  an  operator  performance  enhancement  for  the 
acquisition  and  identification  of  time-critical-targets  (TCT’s).  Both  the  JTF  questionnaire 
results  and  in-flight  observations  indicated  that  the  integration  of  the  ES  AR/ATR 
capabilities  within  the  JSTARS  platform  is  a  technology  advancement  accepted  by  the 
users  in  this  study.  Such  technology  could  potentially  improve  target  prosecution 
performance.  These  new  capabilities  were  also  reported  to  have  a  significant  effect  on 
increasing  SA  during  the  demonstration  flights  without  increasing  workload.  The 
evaluation  also  demonstrated  the  critical  role  of  eliciting  operator  input  to  aid  in  the 
system  design,  development,  and  evaluation  process. 
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GLOSSARY 


AOT 

Aerial  Observer  Technician 

ATR 

Automatic  Target  Recognition 

CRT 

Cathode  Ray  Tube 

ESAR 

Enhanced  Synthetic  Aperture  Radar 

FOA 

Focus  of  Attention 

HSI 

Human  System  Interface 

IA 

Image  Analyst 

ID 

Identification 

JSTARS 

Joint  Surveillance  Target  Attack  Radar  System 

JTF 

Joint  Test  Force 

RTM 

Rapid  Response  Targeting  Against  Mobile  Ground  Targets 

SA 

Situational  Awareness 

SAR 

Synthetic  Aperture  Radar 

SLD 

Second  Level  Detection 

TCT 

Time  Critical  Target 

TEL 

Transporter/Erector/Launcher 

TMD 

Theater  Missile  Defense 
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APPENDIX  A:  TMD/RTM  AIRCREW  QUESTIONNAIRE 
OPERATOR  INTERFACE  ASSESSMENT  RESULTS1 


Subject  Background  &  Experience 


SUBJECT  BACKGROUND 

Subject 

# 

Rank 

Age 

Crew 

Position 

Current 

Aircraft 

Flight 

Hrs. 

Prior 

Aircraft 

Flight 

Hrs. 

Mission 

Date 

1 

CAPT 

36 

WPNS/SURV 

E8 

500 

E-3 

1300 

? 

2 

SMSGT 

38 

SURV 

E-8 

450 

E-3 

2300 

16  Oct  97 

3 

TSGT 

38 

AOT 

E-8 

350 

E-3 

EC- 130 

2500 

500 

6  Nov  97 

4 

CAPT 

29 

WO 

E-8C 

50 

E-3 

2400 

? 

SUBJECT  SENSOR  EXPERIENCE 

Radar  Type 

Operat 

ional  Activity 

Other  Experience 

IR 

Yrs. 

EO 

Yrs. 

Yrs 

i 

E-8 

2 

N 

NA 

N 

NA 

N 

N 

NA 

NA 

NA 

2 

E-8 

3 

NA 

NA 

NA 

NA 

Y 

Y  (Sim) 

E-3 

ESM 

2 

3 

E-8 

3 

NA 

NA  . 

NA 

NA 

NA 

Y  (Sim) 

E-3 

? 

4 

N 

NA 

N 

NA 

N 

NA 

NA 

NA 

E-3 

FPS  117 

Air  Surv. 
GB  Surv. 

■ 

Operational  TMD  Experience: 

Please  list  the  3-5  primary  tasks  that  you  perform  during  a  typical  TMD  mission  or  recent  TMD 
exercise/demonstration  you  have  participated  in. 

(  n  =  3)  Sub.  #  4  had  not  recently  performed  a  TMD  excercise  or  demo. 

The  three  other  respondents  reported  each  of  the  tasks  below: 

•  target  search 

•  pass  target  information  to  ground  agencies  /  fighters 

•  pass  target  updates 

•  develop  target  pictures  to  send 

•  ID  other  relevant  terrain  features  for  updates 


'Please  Note:  The  aggregated  mean  responses  to  questions  G.  and  H.  in  Section  I.  (in  bold)  are 
reported  in  the  Human  Factors  Evaluation  section  of  the  report  in  Tables  2  and  3. 
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I.  Targeting  Information  /  ATR  Cueing 

Ratings  were  given  on  a  5  point  scale:  1=  Not  Important  5  =  Very  Important 


INFORMATION  TYPE 

Rate  importance  of  the  type  of  information  towards  completion  of  a  typical  mission. 

Subject 

# 

Target 

Description 

Target 

Coordinates 

ATR  Confidence 
Levels 

ATR  Cue 
Boxes 

|pfH9 

Other 

Information 

1 

5 

4 

5 

5 

4 

2 

5 

5 

5 

5 

5 

3 

5 

5 

5 

5 

5 

4 

5 

5 

5 

3 

3 

A.  Did  you  find  the  targeting  information  (e.g.  coordinates,  description)  displayed  useful?  Please  explain. 

Y/N 

Comment 

i 

Y 

Somewhat  useful.  I  didn’t  see  any  coordinates  displayed  on  the  GD  that  would  have  been 
helpful.  The  ID  was  displayed  fine. 

2 

Y 

Very  useful,  but  the  coordinates  for  target  ID  need  to  be  displayed  below  ID  on  the  GD. 

3 

Y 

The  coordinate  display  was  very  useful,  but  you  need  to  have  the  capability  to  display  any 
type  of  coordinate  system  and  have  easy  access  to  make  changes. 

4 

Y 

Coordinates  were  very  useful.  The  description  was  not  extremely  useful.  Use  plain 

English  (i.e.,  TEL  or  SA-x)  If  it’s  a  decoy,  just  say  decoy).  j 

|  B.  Was  any  important  information  not  displayed  or  not  easily  accessible?  Please  explain.  I 

Subject  # 

Y/N 

Comment 

1 

Y 

Target  coordinates.  Recommend  putting  the  confidence  level  on  the  GD  with  the  ID.  The 
level  should  be  something  easy  for  the  operator  to  make  a  targeting  decision  (i.e.,  %  or 
scale). 

2 

Y 

The  display  of  normal  GD  was  difficult  and  caused  loss  of  SA. 

3 

N 

No  comment 

4 

N 

No  comment 

C.  Did  you  find  the  ATR  information  useful?  Please  explain  what  was  useful  or  unnecessary  regarding  the  ATR 
information. 

Subject  # 

Y/N 

Comment 

1 

Y 

The  ATR  cueing  symbology  enabled  operator  to  quickly  find  the  target  and  potential 
targets.  The  resolution  of  the  ESAR  is  extremely  beneficial,  allowing  the  operator  to 
pass  targeting  information. 

2 

Y 

It  provided  good  insight  about  possible  targets  and  types. 

3 

Y 

When  you  get  to  the  3rd  level  of  detection,  need  to  show  a  meaningful  confidence  level  in 
both  info  and  on  GD. 

4 

Y 

Yes,  but  it  should  also  have  a  percent  certainty  (e.g.  TEL  Up  %80). 

Subject  # 

Y/N 

Comment 

1 

N 

I  liked  the  fact  that  the  symbology  was  operator-selectable — symbol,  color,  and 
display /hide  function. 

2 

Y&N 

We  need  to  have  access  to  normal  GD  to  maintain  operational  SA. 

3 

N 

4 

N  1 

umaaEmm 


roblems  in  accessing  /  handling  workstation  information 


Subject  # 

Y/N 

1 

N 

2 

Y 

The  files  listed  for  view  were  in  reverse  order  of  normal  interest.  The  last  file  created 
should  be  the  first  one  available  for  view,  not  the  last. 

3 

Y 

You  need  to  be  able  to  use  ft  or  meters  for  distance  measurements  in  the  GD.  Operator 
specify  canonical  type. 

4 

Y 

The  method  of  accessing  ESAR-8  was  somewhat  painful.  You  shouldn’t  have  to  refresh 
every  time  to  get  the  new  SARs.  Should  be  shown  in  the  TD  automatically. 

F.  What  is  vour  assessment  of  the  value  of  the  ATR  Cues/Symbology  overlaid  on  an  image. 


Subject  #  _ 


1  Significantly  Positive  _ 


2  Significantly  Positive _ 


3  Significantly  Positive  _ _ 


Somewhat  Positive 


G.  Rate  Impact  that  the  following  information  had  on  your  SA. 


4  =  Somewhat  Positive  5  =  Significantly  Positive 


ATR  Cue  Boxes 


Subject 

# 


1 


2 


3 


IjEHMRBiifi 


Target 

Coordinates 


ATR  Results 
Summary 


5 


ATR  Box 
Color  Coding 


5 


5 


5 


H.  How  was  your  WORKLOAD  affected  by  receiving  the  following  types  of  information: 


2  =  SW  Increased  3  =  Same  4  =  SW  Decreased  5  =  Signif.  Decreased 


ATR  Cue  Boxes 


Subject 

# 


1 


2 


3 


EgggjTgS 


Target 

Coordinates 


ATR  Results 
Summary 


2 


2 


N/A 


3 


How  would  you  improve  the  application  of  the  ATR  information  to  workstation  display? 

1:  I  would  like  to  see  the  results  incorporated  into  the  normal  Joint  STARS  Graphics 
Display,  instead  of  being  a  separate  display.  This  would  allow  an  operator  to  have  all 
available  information  about  a  target  (i.e.,  Threat  information.  Order  of  Battle,  and  other 
Joint  STARS  information)  on  one  integrated  display.  This  would  increase  an  operator’s 
SA  and  decrease  workload  by  not  having  to  switch  between  windows  (displays). 

Would  like  to  have  a  simple  "confidence  level"  or  factor  displayed  along  with  the 
ATR  cue  boxes.  Something  that  the  operator  could  use  to  determine  the  confidence  that 
the  algorithm  has  correctly  identified  the  target.  I  believe  this  will  be  necessary  in  making 
targeting  decisions.  It  must  be  simple,  like  60, 70,  80%,  or  a  numerical  scale. 

2:  It  needs  to  be  implemented  into  the  normal  ground  display  to  allow  continuous  SA. 
Otherwise,  this  becomes  one  person’s  only  task. 

3:  It  needs  to  be  integrated  somehow  into  the  system,  not  a  stand  alone  system. 

4:  See  previous  comments.  Put  a  simple  Bearing/Range  switch  action  in  the  window 
when  using  ATR. 


II.  ESAR  E-4  &  E-8  Imagery 


|  A.  Did  you  use  the  E-4  SAR  imagery  during  the  mission?  { 

Subject  # 

Y/N 

Comment 

1 

N 

Lab  only. 

2 

Y 

Little  better  detail  in  area  viewed. 

3 

N 

NA 

4 

N 

NA 

B.  What  is  your  assessment  of  the  value  of  the  ESAR  E-4  image  capability  in  aiding  target  acquisition? 

Subject  # 

1 

NA 

2 

Somewhat  Positive 

3 

NA 

4 

NA 

C.  Did  you 

find  the  ESAR  E-8  imagery  useful?  ! 

Subject  # 

Y/N 

1 

Y 

Better  resolution  than  normal  SAR  significantly  increases  SA. 

2 

Y 

The  definition  of  these  images  was  enhanced  and  provided  increased  confidence  of 
possible  IDs  provided. 

3 

Y 

There  was  much  more  detail  than  normal  SAR  imagery. 

4 

Y 

It  provides  a  higher  resolution  picture. 
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D.  Did  the  E-8  imagery  significantly  affect  your  visual  fatigue? 

Subject  # 

Y/N 

1 

Lab  demo. 

2 

N 

It  didn’t  cause  any  more  visual  fatigue  than  viewing  other  SAR  images. 

3 

N 

No  more  than  normal  SAR. 

4 

N 

E.  What  impact  did  the  ESAR  E-8  image  have  on  your  SA? 


Subject  #  _ 


1  Significantly  Positive  _ 


2  Somewhat  Positive  _  _ 


3  Significantly  Positive.  If  ESAR  8  were  integrated  into  the  GD,  definitely  sig.  positive. 


Somewhat  Positive  _ 


F.  Were  there  any  difficulties  working  the  ESAR  E-8  imagery  on  the  JSTARS  workstation? 


Subject#  Y/N  _  _ _ 


1  Lab  _ 


2  Y  See  comments  above.  _ 


Viewed  in  lab.  _ 


No 


G.  Was  the  E-8  image  useful  in  target  acquisition  and  identification? 

Subject  #  Y/N 

1  Y 

Very  useful  in  acquisition.  I  don’t  have  any  training  or  enough  experience  using  the 
imagery  in  identification. 

2  Y 

Provided  increased  visual  ability  on  the  OWS.  Provides  additional  confidence  in  ID 
provided  and  forwarded  to  other  agencies. 

3  Y 

Yes,  with  the  ATR  functions. 

4  Y 

Yes,  if  the  ATR  program  works  when  it  helps  battle  managers  pick  out  valid  targets  on 
battlefield. 

III.  Human  Machine  Interface 


|  A.  Did  you  find  the  Human  Machine  Interface  “user  friendly”? 

Subject  # 

Y/N 

1 

Y 

See  ADDITIONAL  COMMENTS 

2 

N 

Problems  with  the  list,  as  described  earlier.  Loss  of  display  when  trying  to  view  normal 
GD.  More  operator  input  is  required,  if  future  implementation  will  occur. 

3 

Y 

4 

N 

Didn’t  like  the  fact  that  you  had  to  refresh  to  get  new  SARs  in  the  TD. 

17 


B.  What  impact  did  the  Human  Machine  Interface  have  on  your  workload? _ 

Subject  # 

1  _ None _ _ _ 

2  Somewhat  Negative.  Increased  actions  based  on  SA  required. 

3  _ Somewhat  Positive 

4  _ Somewhat  Negative.  See  above _ _ _ 

C.  Please  describe  any  image  handling  or  targeting  operations  that  were  easier  to  perform  using  the  new  interface: 

Subject  #  _ _ _ 

1  No  comment 

2  _ Greater  degree  of  confidence  in  ID  presented. _ 

3  Definitely  enhances  the  operators  ability  to  interpret  the  image;  by  allowing  more  detail. 

4  NA  _ 


D.  Please  describe  any  image  handling  or  targeting  operations  that  were  more  difficult  to  perform  using  the  new 
interface: 

|  Subject  # 

i 

NA 

2 

See  response  in  III  A. 

3 

When  you  crop  out  a  portion  of  the  image  to  save,  on  that  image  we  lost  the  Lat/Long 
position.  It  was  represented  by  6  digits. 

4 

NA 

F.  What  is  your  assessment  of  the  value  of  the  Human  Machine  Interface  to  the  operation  of  the  new  system 

capabilities? _ _ _ 

Subject  # _ _ _ 

1  _ Somewhat  Positive  _ 

2  _ Somewhat  Negative  to  None _ 

3  Somewhat  Positive _ _ 

4  Somewhat  Negative _ _ _ 


**  Please  add  any  Additional  Comments  you  have  concerning  the  ES A R/ ATR/MTI  system: 

Sub  1:  Would  like  to  see  it  incorporated  if  possible.  If  not,  provide  the  confidence  of  the  ID  as 
described  earlier  Also,  display  coordinates  below  the  ID.  The  coordinates  must  be  operator 
selectable  in  the  system,  i.e.  Lat/Long,  UTM,  or  Georef.  Lat/Long  in  either  DMS  or  DMM. 
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Sub  2:  Good,  but  there  are  areas  that  could  definitely  be  improved  for  the  operator. 

Sub  3:  Integration  of  the  ESAR  8  image  and  ATR  products  into  the  GD  would  definitely 
enhance  SA  (by  overlaying  these  products  on  background  info. — i.e.,  maps,  carto,  etc.).  See 
Back. . .  [MISSIN  G] 

Sub  4:  No  additional  comments. 
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